The rise of print and display devices ranging from tiny “thumbprints” of images often seen in selection menus, small, low resolution mobile telephone screens, slightly larger PDA screens, to large, high resolution elongated flat panel display and projector screens has made image resizing an important technique for rendering and viewing digital images. Resizing images to render them on different devices than originally intended is sometimes called image retargeting.
Conventional image retargeting typically involves image scaling and cropping. Image scaling magnifies or shrinks the size of the image to resize the image. Generally, the same scale factor is applied in both the horizontal and vertical directions, which preserves the aspect ratio of the image. Image scaling alone does not work well when the aspect ratio of the image needs to change, because applying different scale factors in the horizontal and vertical directions introduces visual distortions.
Cropping is another method to resize an image by cutting out a subset of pixels within the image. Generally, image scaling is combined with cropping when the aspect ratio of an image needs to be changed. In this case, the image is scaled so that it has the right size in one dimension, but is oversized in the other direction. The scaled image is then cropped to obtain an output image of the desired size.
Many resizing algorithms default to cropping the output image from the central portion of the input image, discarding equal portions of the input image on both edges. However, this can result in discarding important parts of the image depending on the content of the image. While cropping an image, there is a desire to maintain important features in the content of the image. This can be done with top-down or bottom-up methods. Top-down methods use tools such as face detectors to detect important regions in the image, whereas bottom-up methods rely on visual saliency methods to construct visual saliency map of the source image. After the saliency map is constructed, cropping can be used to display the most important region of the image.
One method described by Suh et al., in the article “Automatic thumbnail cropping and its effectiveness” (Proceedings of the 16th annual ACM symposium on User Interface Software and Technology, pp. 95-104, 2003) automatically generates thumbnail images based on either a saliency map or the output of a face detector. With this method, a source image is cropped to capture the most salient region in the image.
Another method taught by Chen et al. in the article “A visual attention model for adapting images on small displays” (Multimedia Systems, Vol. 9, pp. 353-364, 2003) adapts images to mobile devices. In this method, the most important region in the image is automatically detected and transmitted to the mobile device.
Santella et al., in the article “Gaze-based interaction for semiautomatic photo cropping” (ACM Human Factors in Computing Systems, pp. 771-780, 2006), which is incorporated herein by reference, use eye tracking, in addition to composition rules to crop images intelligently. In this method, a users looks at an image, while eye movements are recorded. The recordings are used to identify important image content, and can then automatically crop the image to any size or aspect ratio.
All of the above rely on conventional image resizing and cropping operations to retarget of the image. These approaches are limited because it can only remove pixels from the image periphery. In some cases, there may be important image content at the edges of the image that will be lost during the cropping operation no matter how the image is cropped. More effective resizing can only be achieved by considering the image content as a whole, in conjunction with geometric constraints of the output device.
Another method taught by Gal et al. in the article “Feature aware texturing” (Proc. Eurographics Symposium on Rendering, 2006) uses a feature-aware texture mapping that warps an image to a new shape, while preserving user-specified regions. This is accomplished by solving a particular formulation of the Laplace editing technique suited to accommodate similarity constraints in images. However, local constraints are propagated through the entire image to accommodate all constraints at once, and may sometimes fail.
Another method taught by Agarwala et al. in the article “Interactive digital photomontage” (ACM Trans. Graph. Vol. 23, pp. 294-302, 2004) composes a novel photomontage from several images. A user selects ROIs from different input images, which are then composited into an output image.
One rather elegant content-aware image retargeting algorithm called “seam carving” has been described by S. Avidan and A. Shamir in U.S. Patent Application Publication 2008/0219587, entitled “Method for retargeting images.” The seam carving technique provides a way to systematically remove pixels from visually “unimportant” paths (“seams”) through an image, effectively reducing the height or width by one pixel at a time, in a relatively unnoticeable way. Similarly, pixels can be added to these paths to achieve an increase in the dimension. However, this approach fails if seam passes through the important objects in the image.
Another method using mesh parameterization has been described by Y. Guo et al. in the article “Image retargeting using mesh parameterization,” (IEEE Transactions on Multimedia, Vol. 11, pp. 856-867, 2009). In this approach, a mesh image representation that is consistent with the underlying image structures is constructed for image retargeting. This technique requires processing an entire image at once which may be too complex and too costly for many applications.
Another method described by D. Simakov, et al. in the article “Summarizing visual data using bidirectional similarity” (Proc. IEEE Conference on Computer Vision and Pattern Recognition, pp. 1-8, 2008) uses a similarity measure. In this approach, an image similarity measure is optimized for image retargeting. This technique requires processing an entire image at once which may be too complex and too costly for many applications.
Thus, there exists a need for content-aware image retargeting that preserves salient features of an image even under arbitrary changing of the aspect ratio.